Method and apparatus for low-light imaging enhancement

ABSTRACT

A device and methods are provided for low-light imaging enhancement by a digital imaging device. In one embodiment, a method includes detecting an image associated with ambient lighting of a scene, detecting an image associated with artificial lighting of the scene, aligning the image associated with ambient lighting relative to the image associated with artificial lighting based on a motion parameter of ambient lighting image data and artificial lighting image data, and calculating data for a combined image based on aligned ambient lighting image data and artificial lighting image data, wherein image data for the combined image is selected to maximize an objective quality criterion of the combined image. The method may further include determining an image parameter based on ambient lighting image data, blending image data associated with the artificial lighting and the combined image based on the image parameter to generate a tone rendered image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.61/306,893, filed Feb. 22, 2010.

FIELD OF THE INVENTION

The present invention relates in general to methods for imaging and inparticular to enhancing low-light image capture.

BACKGROUND

Conventional methods for image capture in low-light typically employ oneor more corrective measures to properly detect a scene. For example,handheld cameras typically apply high gain to image data captured of alow-light scene. Applying high gain, however, amplifies electronic andphotonic noise in additional to amplifying the image signal. As aresult, images captured in low-light can appear noisy and contain lesshigh frequency content (e.g., details) due to amplification of thenoise. One conventional solution is to introduce additional light to ascene by activating a flash of the camera during capture. This willresult in a brighter scene, and can allow for gain to be lowered forcapturing image data. However, images captured with flash can produce aharsh flat appearance resulting from the direct lighting. Additionallyflash can change the detected color of a scene and cause near objects toappear significantly brighter than far objects. As a result, manyartifacts not visible to the naked eye, can appear in image datacaptured for the scene.

Another approach is to utilize alternative flash arrangements, such as adiffuse flash or indirect flash, to soften the appearance of artificiallight. Flash warming filters may be applied to modify the color of theflash's light. However, these solutions are not well suited for compactimaging devices, like portable cameras only having a single flash,wherein the flash is usually embedded in the body and is not adjustable.

Another approach employs digital processing of image data. For example,the digital processing approaches described in Digital Photography withFlash and No-Flash Image Pairs, Petschnigg et al., 2004, and PhotographyEnhancement via Intrinsic Relighting, Eisemann and Durand, 2004,digitally separate the flash and ambient images to light and detaillayers (using adaptive filtering tools like the bilateral filter). Thesemethods, however, take details undiscernibly from the image dataresulting in the introduction of additional digital noise to the scenein parts of where the flash image is very dark or (conversely)over-saturated. Accordingly, there exists a desire to correct one ormore of the aforementioned drawbacks.

BRIEF SUMMARY OF THE INVENTION

Disclosed and claimed herein are a device and methods for low-lightimaging enhancement by a digital imaging device. In one embodiment, amethod includes: detecting an image, by the digital imaging device,associated with ambient lighting of a scene, detecting an image, by thedigital imaging device, associated with artificial lighting of thescene, aligning the image associated with ambient lighting relative tothe image associated with artificial lighting based on a motionparameter of ambient lighting image data and artificial lighting imagedata, and calculating data for a combined image based on aligned ambientlighting image data and artificial lighting image data, wherein imagedata for the combined image is selected to maximize an objective qualitycriterion of the combined image. The method may further includedetermining an image parameter based on the ambient lighting image data,blending image data associated with the artificial lighting image andthe combined image based on the image parameter to generate a tonerendered image, and storing, by the digital imaging device, the tonerendered image.

Other aspects, features, and techniques of the invention will beapparent to one skilled in the relevant art in view of the followingdetailed description of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, objects, and advantages of the present invention willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify correspondingly throughout and wherein:

FIG. 1 depicts a simplified block diagram of an imaging device accordingto one embodiment of the invention;

FIG. 2 depicts a process for image enhancement by an imaging deviceaccording to one or more embodiments of the invention;

FIG. 3 depicts a process for image enhancement by an imaging deviceaccording to another embodiment of the invention;

FIG. 4 depicts a process for aligning image data according to oneembodiment of the invention;

FIG. 5 depicts a graphical representation of combining image dataaccording to one embodiment of the invention;

FIG. 6 depicts a graphical representation of approximatingsignal-to-noise ratio according to one embodiment of the invention; and

FIG. 7 depicts a graphical representation of blending image dataaccording to one embodiment of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Overview and Terminology

One aspect of the present invention relates to enhancing low-light imagedata capture. In one embodiment, process is provided for enhancementincluding detecting image data associated with ambient lighting of ascene, and detecting image data with artificial illumination or lighting(e.g., flash) of a scene. The process may include alignment of thecaptured image data, and calculation of a composite image. The processmay further include determining an image parameter, such as a compositemask, and blending image data associate with image data captured withartificial light, and the composite image based on an image parameter,such as a masking image. As such, details may be selected from one ormore of the image data captured using ambient light and image datacaptured based on artificial illumination. In addition, tone renderedblending of composite image data may be employed to optimize combinationof image data.

In another embodiment, an imaging device is provided to capture of imagedata associated with ambient light and artificial light to enhancecaptured image data. The imaging device, such as a digital camera, maybe configured to determine one or more corrections for image data.

As used herein, the terms “a” or “an” shall mean one or more than one.The term “plurality” shall mean two or more than two. The term “another”is defined as a second or more. The terms “including” and/or “having”are open ended (e.g., comprising). The term “or” as used herein is to beinterpreted as inclusive or meaning any one or any combination.Therefore, “A, B or C” means any of the following: A; B; C; A and B; Aand C; B and C; A, B and C. An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

Reference throughout this document to “one embodiment”, “certainembodiments”, “an embodiment” or similar term means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the presentinvention. Thus, the appearances of such phrases in various placesthroughout this specification are not necessarily all referring to thesame embodiment. Furthermore, the particular features, structures, orcharacteristics may be combined in any suitable manner on one or moreembodiments without limitation.

In accordance with the practices of persons skilled in the art ofdigital image processing and analysis, the invention is described belowwith reference to operations that can be performed by a computer systemor a like electronic system. Such operations are sometimes referred toas being computer-executed. It will be appreciated that operations thatare symbolically represented include the manipulation by a processor,such as a central processing unit, of electrical signals representingdata bits and the maintenance of data bits at memory locations, such asin system memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits

When implemented in software, the elements of the invention areessentially the code segments to perform the necessary tasks. The codesegments can be stored in a “processor storage medium,” which includesany medium that can store information. Examples of the processor storagemedium include an electronic circuit, a semiconductor memory device, aROM, a flash memory or other non-volatile memory, a floppy diskette, aCD-ROM, an optical disk, a hard disk, etc.

Exemplary Embodiments

Referring now to the figures, FIG. 1 depicts a simplified block diagramof an imaging device according to one embodiment of the invention. Asdepicted imaging device 100 includes processor 105 coupled to imagesensor 110, memory 115, and display 120. Image sensor 110 may relate toa combination of an optical lens and light detection circuitry (e.g.,CMOS integrated circuit, charge coupled device (CCD), etc.). Image datadetected by image sensor 110 may be provided to processor 105, includingimage data detected for generating a preview image.

Processor 105 may be implemented using one of an integrated circuitmicroprocessor, microcontroller, digital signal processor and processorin general. Processor 105 may be configured to process received imagedata based on, for example, specific image processing algorithms storedin memory 115 in the form of processor-executable instructions. Inaddition, processor 105 may be configured to control exposureparameters, such as exposure periods, focus depth, etc. In certainembodiments, processor 105 may be configured to control one or morecomponents of imaging device 100 including image sensor 110 andillumination source 125. Illumination source 125 may relate to anartificial illumination source, such as an electronic flash integratedwith imaging device 100.

In on embodiment, image data processed by processor 105 may be stored inmemory 115 and/or provided to display 120 for viewing. It should beappreciated that memory 115 may relate to any combination of differentmemory storage devices, such as a hard drive, random access memory(RAM), read only memory (ROM), flash memory, or any other type ofvolatile and/or nonvolatile memory. It should further be appreciatedthat memory 115 may be implemented as multiple or discrete memories forstoring processed image data, as well as the processor-executableinstructions for processing the captured image data. Further, memory 115may include removable memory, such as flash memory, for storage of imagedata.

Display 120 may relate to a liquid crystal display (LCD) incorporatedinto digital camera 100 for display of captured image data.Alternatively, it should be appreciated that an external display devicemay be coupled to imaging device 100 for display.

According to one embodiment of the invention, imaging device 100 may beconfigured to enhance image data captured in low-light. In addition tothe conventional methods for generating artificial light, (e.g., aflash), imaging device 100 may be configured to enhance image data basedon image data captured for a scene associated with ambient light and/orartificial light generated by illumination source 125. Illuminationsource 125 may be configured to emit a short burst of light based onuser activation of the Input/Output (I/O) 130 (e.g., user triggering ofa capture button). Light emitted by illumination source 120 mayilluminate a scene for the fraction of a second during image capture. Itmay also be appreciated that illumination source 120 may be configuredto provide a wide variety of flash signals, including one or more burstsof light associated with one or more output levels. I/O 130 may relateto one or more buttons or terminals of the imaging device for adjustinguser settings and/or communication data by the imaging device.

Although FIG. 1 has been described above with respect to an imagingdevice, it should be appreciated that the device may relate to otherdevices, and/or be included in other devices, such as a mobilecommunication device and portable communication device in general, etc.

Referring now to FIG. 2, a process is depicted for low-light imageenhancement by an imaging device according to one or more embodiments ofthe invention. Process 200 may be performed by one or more elements ofthe device of FIG. 1 (e.g., imaging device 100), such as a processor(e.g., processor 105). Process 200 may employed for enhancing image datadetected for low-light image capture. As depicted in FIG. 2, process 200may be initiated by detecting image data at block 205. In oneembodiment, image data is detected by the imaging device associated withthe ambient light of a scene (e.g., no artificial light, or flash,generated by the imaging device). Ambient image data detected by theimaging device may relate to image data detected by the imaging deviceto provide a preview image for the user via a display (e.g., display120). Image data detected by the imaging device may additionally includedetection of image data associated with artificial lighting of a scene,such as a flash emitted by the imaging device. In one embodiment, imagedata associated with ambient lighting of the scene is captured prior tocapture of image data with artificial illumination. However, it may alsobe appreciated that image data detect at block 205 may include imagedata associated with ambient lighting captured following image capturewith artificial illumination. As will be discussed below, image datadetected by the imaging device of a scene and utilized by process 200may be based on two images, a first associated with ambient lighting andsecond associated with artificial illumination (e.g., flash). It shouldalso be appreciated that process 200 may include detection of additionalimages of a scene. For example, third and fourth images, associated withambient lighting and artificial lighting, respectively may be detectedas will be described below in more detail with respect to FIG. 3.

In contrast to conventional methods of low-light image enhancement whichdo not align image data due to complexity of image data with variedillumination levels and noise level of ambient image data, alignment ofthe image data at block 210 may be provided according to one embodiment.At block 210, image data detected by the imaging device may be alignedto allow for disparities in image data to be accounted for due to delayin capturing image data. In one embodiment, aligning may relate toaligning a first image relative to a second image, wherein the firstimage relates to image data captured with ambient lighting and thesecond image relates to image data detected with artificial illuminationby the imaging device. Image alignment may be based on a motionestimation algorithm such as the motion estimation algorithm methodsdescribed in commonly assigned patent publication no. US 2008/0291330 A1titled Technique of Motion Estimation When Acquiring An Image of A SceneThat May Be Illuminated With A Time Varying Luminance, which is herebyfully incorporated by reference. In one embodiment the motion estimationalgorithm employed for image alignment at block 210 may relate to motionestimation invariant to scene illumination to align frames. In certainembodiments, the imaging device may provide an indication when alignmentof image data may not be provided as will be described below in moredetail with reference to FIG. 3.

At block 215, a combined image may be calculated based on the detectedimage data. One advantage of the invention may be to utilize detailsfrom image data of either of the ambient image and illuminated/flashimage, in contrast to conventional methods which only utilize imagedetected with flash. As will be discussed in more detail below withrespect to FIG. 5, combined image data may be calculated to create anoptimal combined image to include illumination details associated withimage data of the ambient image and illuminated image, and to maximizedetail and minimize noise. In one embodiment, image data for thecombined image is selected to maximize an objective quality criterion ofthe combined image. For example, the objective quality criterion mayrelate to signal to noise ration.

In certain instances, combined, image data may appear unnatural or flat.Accordingly, combined image data may be blended with image dataassociated with artificial illumination to generate a continuous flasheffect in one embodiment. For example, enhancement of image data tocorrect for a flat or unpleasant appearance of artificial light on imagedetails. In one embodiment, process 200 may include determine acompositing mask at block 220 to blend image data associated with theilluminated image with combined image data at block 225. In order toselectively correct for unpleasing areas of image data, portions of theimage data which do not have a pleasing appearance may be replaced togenerate an optimal tone rendered result at block 225. Thus, blendingimage data at block 225 can localize the correction only to requiredregions. Based on the blending of image data, the digital imaging devicemay store a tone rendered result at block 230.

Referring now to FIG. 3, a process is depicted for low-light imagingenhancement according to another embodiment. Process 300 may beperformed by a processor of an imaging device (e.g., imaging device 100)according to one embodiment. Process 300 may allow for abortingenhancement of image data when alignment of detected images may not beperformed, or when said alignment process fails. Process 300 may beinitiated by detecting image data at block 305, such as image dataassociated with ambient light of a scene and image data associated withartificial lighting of a scene, such as a flash emitted by the imagingdevice. At block 310, the processor of the imaging device may alignimage data. The processor may then check the alignment of the image dataat decision block 315. When alignment of the image data produces analignment error (e.g., “YES” path out of decision block 315) process 300may abort the enhancement of detect image data at block 320. Forexample, the process 300 processor may abort image enhancement when itis detected that significant areas of the image exhibit localmisalignment as will be discussed in more detail below with respect toFIG. 4. In certain embodiments, process 300 may display a message to auser of the imaging device that captured image data resulted in anerror. Alternatively, one of the source images (such as the flashimage), may substitute the result of the process 330 in these instances.Alternatively, or in combination, the imaging device may prompt the userto capture another image of the scene. Accordingly, process 300 maycapture image data as depicted at block 305, such as third and fourthadditional frames.

When image data does not result in an alignment error (“NO” path out ofdecision block 315), process 300 may then create a combined image atblock 325. Process 300 may then calculate a tone rendered image at block330 based, at least in part, on image data detected at block 305.

Referring now to FIG. 4, a process is depicted for detecting alignmenterrors according to one embodiment. Because image data employed forimage enhancement (e.g., image data detected at blocks 205 and 305) isnot captured at the same instant, motion of the imaging device, such asuser hand shake, can cause undesirable artifacts in combined image data.In one embodiment, image data is aligned by calculating motion of imagecontent between frames. Inverse motion may be applied to one image framein order to compensate for motion according to one embodiment.

Local alignment errors may be caused when an object in a scene is movingquickly. In certain instances, alignment of image data for globalmotion, such as camera shake, may not compensate for local movement.According to one embodiment, local motion may be detected by searchingfor an area of the image which includes motion after compensation forglobal motion. When significant areas in the image exhibit localmisalignment, enhancement of image data may be aborted or the region ofimage data may be replaced. Process 400 provides for estimation ofmotion and a global transformation. In one embodiment, process 400 mayestimate global transformation based on a statistical estimationalgorithm, such as Random Sample Consensus Algorithm (RANSAC).

Process 400 may be initiated by determining one or more motion vectorsat block 405. Motion vectors may be determined for a plurality of pointsin the image. In one embodiment, motion vectors may be randomlyselected. By way of example, a set of point and motion vectors detectedfor a point in first and second images (e.g., ambient and illuminatedimages) may be characterized as:{(x _(i) ,y _(i) ,u _(i) ,v _(i))}_(i=1) ^(N)wherein (x_(i),y_(i)) relates to the point in coordinates matching thepoint in coordinates (x_(i)+u_(i), y_(i)+u_(i)) in the ambient frame.

From a set of correspondences, a global estimation may be determined atblock 410. In one embodiment, the global transformation may be anestimation such as:T _(global):

→

In one embodiment, the global transformation may be applied to everypoint in the artificially illuminated frame. Using a robust statisticsalgorithm, such as RANSAC, motion vectors which do not agree with themajority model may be rejected resulting in a set of indices accordingto the following:G _(outliers) ⊂{1,2, . . . ,N}wherein motion vectors with indices in the set do not agrees with themajority model T_(global).

A new transformation may then be determined, wherein the points inG_(outliers) may be registered. The group of outliers that do not agreewith the model be characterized as:G _(outliers) ^(local) ⊂G _(outliers)

When the model is statistically significant, the group would berelatively large. Based on this model, the alignment may be considered afailure due to local motion when:|G _(outliers) ^(local) |/|G _(outliers)|>θfor some predetermined threshold θ, which is set by observing the ratioin known data (e.g., training). According to one embodiment, in thatstate, a misalignment error may be detected.

Accordingly, process 400 may check for a misalignment error at block415. When a misalignment error is not detected (e.g., “NO” path out ofdecision block 415), process 400 may proceed with enhancement of imagingdata at block 420. When a misalignment error is detected (e.g., “YES”path out of decision block 415), a result may be determined to abortenhancement of image data at block 425.

Referring now to FIG. 5, a process is depicted for calculating acombined image according to one embodiment of the invention. Process 500may be employed to combine image data associated with a first imageassociated with an ambient light level, and a second image associatedwith artificial light emitted by the imaging device during capture.Process 500 may begin by receiving image data, depicted as 505 a and 505b, for an image associated with artificial illumination and image dataassociated with ambient light, respectively. Received image data, 505 aand 505 b, may be filtered by filters 510 and 515. In one embodiment,filters 510 and 515 relate to edge preserving smoothing filters (EPSF).Ambient image values (e.g., L_(ambient)) and the flash image values(e.g., L_(flash)), may be filtered to produceS_(ambient)=Filter{L_(ambient)}, and S_(flash)=Filter{L_(ambient)}.

According to one embodiment, digital image acquisition for a capturedpixel response L from pixel illumination E, can be expressed as:L=E+n(0,E)+n(0,σ_(D) ²)wherein n(m,s) is a stochastic process with variance s and mean m.n(0,E) relates to the shot noise which is proportional to the root ofthe signal level, while (n, σ² _(D)) is sensor noise, or dark noisewhich is not dependent of the signal level E.

For many imaging devices, analog gain G is applied to the readout Lbefore it is digitized as follows;L=(E+n(0,E)+n(0,σ_(D) ²))·G

Defining illumination S as S=EG, the output may be characterized asfollows:

$L = {\left( {\frac{S}{G} + {n\left( {0,\frac{S}{G}} \right)} + {n\left( {0,\sigma_{D}^{2}} \right)}} \right) \cdot G}$

In order to establish a statistical framework, the signal may berepresented as a random signal n(S, F(S)):

$L = {\left( {\frac{n\left( {S,{F(S)}} \right)}{G} + {n\left( {0,\frac{S}{G}} \right)} + {n\left( {0,\sigma_{D}^{2}} \right)}} \right) \cdot G}$Wherein F(S) is the desired detail layer in the image, it is thedependent on the statistics of each image. However, any F function mustfollow the constraints: F(0)=0, as a zero signal has zero details, dueto the quantization limit. F(1)=0 as a saturated signal also cannot holdany detail, that is the saturation limit.

The variance of the captured pixel response, assuming that an imagelevel S the different noise component and the image details areindependent can be referred to as:

${{var}\; L} = {\underset{signal}{\underset{︸}{F}(S)} + \underset{\underset{noise}{︸}}{{GS} + {G^{2}\sigma_{D}^{2}}}}$

Signal to noise ratio (SNR) is a good metric of perceptual quality ofthe detail layer. Based on the above-identifier variance formulation,SNR may be characterized as follows:

${SNR}\overset{\Delta}{=}{\frac{\sigma_{signal}}{\sigma_{noise}} = \sqrt{G^{- 1}\frac{F(S)}{S + {G\;\sigma_{D}^{2}}}}}$wherein G and σ_(D) are known parameters, and F(S) may be experimentallydetermined. Accordingly, the SNR quotient function may be characterizedas follows:

$\begin{matrix}{{{qSNR}\left( {S_{flash},S_{ambient}} \right)} = {\frac{{SNR}_{flash}\left( S_{flash} \right)}{{SNR}_{ambient}\left( S_{ambient} \right)} =}} \\{= {\sqrt{\frac{G_{ambient}}{G_{flash}} \cdot \frac{S_{ambient} + {G_{ambient}\sigma_{D}^{2}}}{S_{flash} + {G_{flash}\sigma_{D}^{2}}} \cdot \frac{F\left( S_{flash} \right)}{F\left( S_{ambient} \right)}}.}}\end{matrix}$

In one embodiment, an optimal flash and non-flash image may be composedby selecting a detail layer based on the qSNR function. By way ofexample, for each pixel qSNR may be determined wherein detailreplacement will be carried out when the value is greater than 1, anddetail replacement may be bypassed and the pixel will keep ambientdetails. One advantage of calculating a qSNR may be to detect artifacts.Shadows may be efficiently removed in particular areas where the SNR ofthe ambient image is larger than the SNR of the flash image. Flashspeculiarities will be removed since, in those areas, the flashillumination level nears saturation and F(.) (hence, flash SNR) drops tozero

According to another embodiment, the qSNR function may be optimized asthe function may be difficult to evaluate in a real-time environment. Incertain instances, the qSNR function may not be particularly meaningful,only whether the value is below or above the value of 1. For example, asexpressed in the iSNR function:

${{iSNR}\left( {S_{flash},S_{ambient}} \right)} = \left\{ \begin{matrix}1 & {{{qSNR}\left( {S_{flash},S_{ambient}} \right)} > 1} \\0 & {otherwise}\end{matrix} \right.$

One solution of the invention includes that the iSNR function can, inturn be approximated. For example, multiple threshold linear functionsmay be employed. A linear threshold function (LIN) may relate to a3-vector (a,b,c) function as follows:

${{LIN}\left( {S_{flash},S_{ambient}} \right)} = \left\{ \begin{matrix}0 & {{{a \cdot S_{flash}} + {b \cdot S_{ambient}} + c} > 0} \\1 & {otherwise}\end{matrix} \right.$

One advantage of the linear threshold function may be the ability toevaluate in real-time. The iSNR approximation may thus be characterizedas:

${{iSNR}\left( {S_{flash},S_{ambient}} \right)} = {\overset{M}{\bigcup\limits_{i = 1}}{\underset{j = 1}{\bigcap\limits^{N_{i}}}{{LIN}_{i,j}\left( {S_{flash},S_{ambient}} \right)}}}$Wherein the union and intersection operators represent logical OR andAND operators respectively (e.g., 1 OR 0=1, 0 OR 0=0). iSNR function 520of FIG. 5 may relate to the above formulation. As will be discussed inmore detail below, FIG. 6 depicts a graphical representation ofapproximating the iSNR function.

Combiner 525 may be configured to combine image data as processed by theiSNR function in combination with image data associated with a flashimage 505 a and ambient image 505 b. Combination of image data may bebased on details of the flash image and/or ambient image data. Acombined image may be output via output 530. As will be discussed below,image data 530 may further include output of a tone rendered result.

According to another embodiment, perceived brightness of a real worldobject can generally be characterized as:L=S·Rwherein S relates to the illumination (which depends only on the lightsource) and R relates to the reflectance (which depends only on theobject being observed).

In one embodiment, illumination may be assumed to be constant, as it isrelatively slow changing in space, when it meets the edges of objects.Accordingly, edge-preserving smoother filtering (e.g., filters 510 and515) may provide the illumination S for an image wherein the extrinsicdetails of an image may be extracted as follows:

$R = \frac{L}{{Filter}\left\{ L \right\}}$

The method of detail replacement in the combined image, of the priorart, is usually performed by:

$L_{combined} = {{\frac{L_{flash}}{{Filter}\left\{ L_{flash} \right\}} \cdot {Filter}}\left\{ L_{ambient} \right\}}$

The aforementioned approach is impractical for some applications, suchas mobile or imbedded implementations. However, applying a logarithmicfunction allows for a linear equation characterized as:L _(initial)=exp[log(L _(flash))−log(Filter{L _(flash)})+log(Filter{L_(ambient)})]

According to one embodiment, each image pixel may be represented by 8bits, and the logarithmic function may be may be compared with alook-up-table to directly translate each of the 256 possible imagesvalues to a logarithmic value. The exponential (exp) function can alsobe evaluated via look-up-table if the base and precision of the logfunction is chosen correctly (e.g., base 2 and precision of 12 bits).The look-up-table may be provided by a processor (e.g., processor 105)of the imaging device. Accordingly, the combined pixel response may bedetermined as follows;

$L_{combined} = \left\{ \begin{matrix}L_{initial} & {{\left( {S_{flash},S_{ambient}} \right)} = 1} \\L_{ambient} & {otherwise}\end{matrix} \right.$

Combined image data based on the above-identified format may correct fornoise and to add missing detail, however may appear to many viewers ashaving an unnatural appearance. Some conventional methods of blendingimage data to obtain a continuous flash effect. However, this approachis non-selective and can result in many regions of the image havingcorrection reduced and/or reproducing image data inaccurately.

According to one embodiment of the invention, and in contrast toconventional methods of blending image data, one or more regions form aflash image may be employed to correct for visually unpleasant regionsof image data in the combined image by combiner 525. For example, areasin which the ambient illumination is very low, and thus unreliable toestimate, may be replaced with data from the flash image. In oneembodiment, the value of each composited pixel may be described as:L _(composited) =L _(flash) M _(compositing) +L _(combined)(1−M_(compositing))Where:M _(compositing) =Q _(ambient)(S _(ambient))·Q _(flash)(S _(flash))

According to one embodiment, the Q functions may be chosen throughsubjective testing to optimize tone rendering for image data. FIG. 7depicts a graphical representation of image blending which may beemployed by combiner 525 according to one embodiment.

Referring now to FIG. 6, a graphical representation is depicted ofapproximating the iSNR function. Graph 605 depicts an iSNR function,wherein dark regions, shown by 610 and 615, relate to a value of 0, andbright regions, shown by 620, relate to a value of 1. Graph 650 relatesto an approximation of iSNR as may be employed for combining image dataaccording to one embodiment of the invention. As depicted, dark regionsof graph 650, shown by 655 and 660, relate to a value of 0, and brightregion, shown by 665, relates to a value of 1. Graph 650 depicts apiecewise linear function containing two groups of two lines eachaccording to one embodiment. It may be appreciated that the approximatediSNR function 650 may be employed for low-light image enhancementaccording to one embodiment of the invention.

FIG. 7 depicts a graphical representation of image blending according toone embodiment. Image blending as depicted in FIG. 7 may be employed bythe combiner of FIG. 5 according to one embodiment. As depicted, a firstimage, ambient image 705 and second image, flash image 710, may beemployed to generate combined image 715. Combined image 715 may relateto a composed image, wherein the image is corrected for artifacts.Compositing mask 720 may be generated based on image data associatedwith the ambient image 705 and a Q function (e.g., Q function describedin reference to FIG. 5). In one embodiment, image data associated withflash image 710 and combined image 715 may be blended based, at least inpart, on compositing mask 720 to generate tone mapped image 725. Incertain embodiments, portions of ambient image, shown as 730, and flashimage, shown as 735 may be employed to correct for one or more portionsof combined image 715 resulting in enhanced image data for a blendedimage, shown as 740 of tone mapped image 725. In that fashion, low-lightimage enhancement may be provided for an imaging device according to oneembodiment.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art. Trademarks and copyrightsreferred to herein are the property of their respective owners.

1. A method for low-light imaging enhancement by a digital imagingdevice, the method comprising the acts of: detecting an image, by thedigital imaging device, associated with ambient lighting of a scene;detecting an image, by the digital imaging device, associated withartificial lighting of the scene; aligning the image associated withambient lighting relative to the image associated with artificiallighting based on a motion parameter of ambient lighting image data andartificial lighting image data; calculating data for a combined imagebased on aligned ambient lighting image data and artificial lightingimage data, wherein the image data of either the aligned ambientlighting image or the artificial lighting image is selected for eachpixel of the combined image to maximize a signal to noise ratio of thecombined image; determining an image parameter based on the ambientlighting image data; blending the image data associated with theartificial lighting image and the combined image based on the imageparameter to generate a tone rendered image; and storing, by the digitalimaging device, the tone rendered image.
 2. The method of claim 1,wherein the image associated with ambient lighting relates to lowerresolution preview image data detected by the imaging device having alower resolution than a resolution of the image associated withartificial lighting.
 3. The method of claim 1, wherein the imageassociated with artificial lighting relates to a full resolution imagewith a lower gain than a gain of a full resolution image associated withambient lighting.
 4. The method of claim 1, wherein the image associatedwith ambient lighting is detected prior to detection of the imageassociated with artificial lighting.
 5. The method of claim 1, whereinthe image associated with artificial lighting is detected prior todetection of the image associated with ambient lighting.
 6. The methodof claim 1, wherein the motion parameter relates to a global alignmentof the ambient lighting image data and artificial lighting image databased on inverse motion.
 7. The method of claim 1, wherein aligning isbased on an alignment parameter of an alignment transformation appliedto image data associated with the regions of interest of the ambientlighting image data.
 8. The method of claim 1, wherein the combinedimage rotates to a composite image based on local illumination of eachpixel.
 9. The method of claim 1, wherein the image parameter relates toa composite mask determined based a tone rendering function.
 10. Themethod of claim 1, wherein blending image data corrects for flatness ofimage data associated with artificial illumination.
 11. The method ofclaim 1, wherein blending relates to blending image data from theartificial lighting image where image data associated with the ambientlighting image is hard to estimate.
 12. The method of claim 1, furthercomprising filtering image data of the ambient lighting and artificiallighting images based on an edge preserving smoothing filter.
 13. Themethod of claim 1, further comprising amplifying image data based onminimal digital gain for a balanced exposure of image data.
 14. Themethod of claim 1, further comprising detecting an alignment error, andaborting enhancement of image data based on the alignment error.
 15. Themethod of claim 14, further comprising capturing an additional imageassociated with ambient lighting of the scene, and an additional imageassociated with artificial lighting of the scene based on the alignmenterror.
 16. A device configured for low-light imaging enhancement, thedevice comprising: an image sensor configured to capture image data ofthe scene; and a processor coupled to the image sensor, the processorconfigured to detect an image associated with ambient lighting of thescene; detect an image associated with a artificial lighting of thescene; align the image associated with ambient lighting relative to theimage associated with artificial lighting based on a motion parameter ofambient lighting image data and artificial lighting image data;calculate data for a combined image based on aligned ambient lightingimage data and artificial lighting image data, wherein the image data ofeither the aligned ambient lighting image or the artificial lightingimage is selected for each pixel of the combined image to maximize asignal to noise ratio of the combined image; determine an imageparameter based on the ambient lighting image data; blend the image dataassociated with the artificial lighting image and the combined imagebased on the image parameter to generate a tone rendered image; andstore the tone rendered image.
 17. The device of claim 16, wherein theimage associated with ambient lighting relates to lower resolutionpreview image data detected by the imaging device having a lowerresolution than a resolution of the image associated with artificiallighting.
 18. The device of claim 16, wherein the image associated withartificial lighting relates to a full resolution image with a lower gainthan a gain of a full resolution image associated with ambient lighting.19. The device of claim 16, wherein the image associated with ambientlighting is detected prior to detection of the image associated withartificial lighting.
 20. The device of claim 16, wherein the imageassociated with artificial lighting is detected prior to detection ofthe image associated with ambient lighting.
 21. The device of claim 16,wherein the motion parameter relates to a global alignment of theambient lighting image data and artificial lighting image data based oninverse motion.
 22. The device of claim 16, wherein aligning is based onthe alignment parameter of an alignment transformation applied to imagedata associated with the regions of interest of the ambient lightingimage data.
 23. The device of claim 16, wherein the combined imagerelates to a composite image based on local illumination of each pixel.24. The device of claim 16, wherein the image parameter relates to acomposite mask determined based a tone rendering function.
 25. Thedevice of claim 16, wherein blending image data corrects for flatness ofimage data associated with artificial illumination.
 26. The device ofclaim 16, wherein blending relates to blending image data from theartificial lighting image where image data associated with the ambientlighting image is hard to estimate.
 27. The device of claim 16, whereinthe processor is further configured to filter image data of the ambientlighting and artificial lighting images based on an edge preservingsmoothing filter.
 28. The device of claim 16, wherein the processor isfurther configured to amplify image data based on minimal digital gainfor a balanced exposure of image data.
 29. The device of claim 16,wherein the processor is further configured to detect an alignmenterror, and abort enhancement of image data based on the alignment error.30. The device of claim 29, wherein the processor is further configuredto detect additional image associated with ambient lighting of thescene, and an additional image associated with artificial lighting ofthe scene based on the alignment error.